refactor(aligned): typed run plan + recorded estimand/population metadata (#394 pillar 4) - #450
Open
ethanbuckley wants to merge 1 commit into
Open
refactor(aligned): typed run plan + recorded estimand/population metadata (#394 pillar 4)#450ethanbuckley wants to merge 1 commit into
ethanbuckley wants to merge 1 commit into
Conversation
…data (#394 pillar 4) Extends the ITT / gain-factor / level-factor / DiD / concurrent run-plan pattern to the per-protocol onset-aligned family. A new aligned.py provides AlignedModelSettings (fail-fast on a misspelled spec.extra key; validates the 4 legacy keys) and a resolved AlignedRunPlan that drives data preparation, factory construction and the config.json / model_recipe.md audit trail, recording the design, estimand, causal status (per-protocol association -- the cohort term is NOT the randomised ITT effect) and analysis population alongside every fit. fit_aligned now resolves and validates the plan before the context resets an output directory or the loader reads data, then loads via plan.prepare_kwargs() (the onset-aligned loader; include_dose requests the session-dose covariate only for the sensitivity variant) and builds via plan.factory_kwargs(). reporting._resolved_run_plan gains an aligned branch so write_model_recipe and config.json's resolved_run_plan pick it up. Byte-identical: the resolved prepare + factory kwargs reproduce the former inline fit_aligned logic exactly for all 9 registered aligned models (verified programmatically). al-001 dev fit is clean end-to-end and now emits model_recipe.md + a populated resolved_run_plan. 12 new run-plan tests; drift suites pass; ruff / format:check / spellcheck clean. Part of #394 (pillar 4: typed settings + resolved plans). No estimand, likelihood, prior, population, fitted equation, sampling preset or artefact schema changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the “typed settings + resolved run plan” pattern to the onset-aligned per-protocol statistical-model family (kind="aligned"), so aligned fits validate settings early (before output reset / data load) and persist a resolved plan + recipe metadata alongside the run for auditability (#394 pillar 4).
Changes:
- Adds
AlignedModelSettingsandAlignedRunPlan, with strict legacyspec.extrakey validation and resolved audit metadata (design/estimand/causal status/population/missingness). - Refactors
fit_alignedto resolve the plan up front, driveload_and_prepare_aligned/build_aligned_modelfrom the plan, and writemodel_recipe.md. - Extends reporting to reconstruct/emit aligned run-plan metadata and adds a registered-spec coverage test suite for aligned models.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/statistical_models/test_aligned_run_plan.py | Adds tests for aligned typed settings, run-plan resolution, and “all registered aligned models resolve with metadata” coverage. |
| src/language_reading_predictors/statistical_models/reporting.py | Adds aligned plan reconstruction so model_recipe.md and config.json.resolved_run_plan are written for aligned fits. |
| src/language_reading_predictors/statistical_models/pipeline.py | Updates fit_aligned to resolve/store the aligned plan before context reset / data load and to drive prepare/factory kwargs from it. |
| src/language_reading_predictors/statistical_models/aligned.py | New aligned-family typed settings + resolved run plan implementation and metadata/recipe generation. |
Comment on lines
+204
to
+215
| design = ( | ||
| "Per-protocol onset-aligned single-gain ANCOVA: a cross-sectional " | ||
| "Beta-Binomial regression of the aligned post-score on its own onset baseline, " | ||
| "age-at-onset and cognitive ability, optionally with a cohort indicator and " | ||
| "the cumulative session dose. One row per child, so no child random intercept." | ||
| ) | ||
| estimand = ( | ||
| "The cohort contrast at the two arms' own onset-aligned endpoints -- a " | ||
| "per-protocol association, NOT the randomised ITT effect (it is confounded by " | ||
| "age-at-onset and cohort/timing). With the dose variant, the cumulative-session " | ||
| "covariate is a collider descendant of group and ability, a sensitivity variant." | ||
| ) |
Comment on lines
+6132
to
+6138
| # Resolve and validate the family contract before the context resets an output | ||
| # directory or the loader reads any data (#394 pillar 4). One plan drives | ||
| # preparation, factory arguments, the teaching recipe and config.json. | ||
| plan = resolve_aligned_run_plan(spec) | ||
| ctx = make_context(spec, config) | ||
| extra = spec.extra | ||
| ctx.resolved_plan = plan | ||
| _report.write_model_recipe(ctx) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Drafted by an LLM-based AI tool (Claude Code/Opus 4.8).
#394 pillar 4 (typed settings + resolved plans), extending the pattern from the merged ITT/gain-factor (#418) and the level-factor (#445) + DiD (#448) + concurrent (#449) run-plans to the per-protocol onset-aligned family (LRPAL). Independent, off
main.What this does
A new
aligned.pygives the family the same typed boundary:AlignedModelSettings— fail-fast on a misspelledspec.extrakey; validates the 4 legacy keys (ability_covariate,use_cohort,use_dose,likelihood).AlignedRunPlan— resolved and validated before the context resets an output directory or the loader reads data; drivesload_and_prepare_aligned+build_aligned_modeland records the design / estimand / causal status / analysis population / missing-data assumption. The estimand prose states plainly that the cohort contrast is a per-protocol association, not the randomised ITT effect (confounded by age-at-onset and cohort/timing), and that the session-dose term is a collider-descendant sensitivity variant.fit_alignedconsumesplan.prepare_kwargs()(the onset-aligned loader;include_doseonly for the dose variant) andplan.factory_kwargs();reporting._resolved_run_plangains analignedbranch so the recipe + config metadata are written.Byte-identical
Verified programmatically: for all 9 registered aligned models, the resolved
prepare_kwargsandfactory_kwargsreproduce the former inlinefit_alignedlogic exactly (loader outcomes/ability/include_dose; factory cohort/dose/likelihood; off-floor node).al-001dev-fits clean end-to-end (52 obs, one row per child, 0 divergences) and now emitsmodel_recipe.md+ a populatedresolved_run_plan.12 new run-plan tests (
test_aligned_run_plan.py); the drift guards pass;ruff/format:check/spellcheckclean.Scope
Mechanical, behaviour-preserving relocation only — no estimand, likelihood, prior, analysis population, fitted equation, sampling preset, diagnostic threshold or artefact schema change (per #394's contract). Pillar-4 families now converted: gain (merged), level, DiD, concurrent, aligned. Remaining: adjusted, growth/historical/measurement (clean single-fit), and mediation (the 3-fit-function outlier, a dedicated multi-PR effort).
Assigning to a human for review and merge.